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Currently, the quality of health services in Indonesia is getting better. 
However, referral systems still face many problems. The first is the difficulty 
in determining the referral hospitals based on patients' background and 
condition. The second is the many different parameters used to determine the 
referral hospitals between countries. Based on these issues, it is necessary to 
improve the current systems’ capabilities. This study develops a model of an 
e-referral information system that takes into account patients’ background. 
Some of the methods used in the model include information technology for 
patient care and Fuzzy Multiple-Criteria Decision Making to determine 
referral hospitals that fit patients’ background. The result of this study is an 
intelligent system model to find referral hospitals that are most suitable to 
patients’ backgound. In order to evaluate the model, we accumulate opinions 


of experts whether the output of the developed model matches the decisions 
taken by the experts. The evaluation indicates that the model obtains a 
conformity level of 81.45%. 
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1. INTRODUCTION 

The development of information, telecommunication, and networking technologies has brought 
changes to health services. Utilization of information and communication technologies in the health service 
systems raises the term e-health. E-health is the use of information and communication technologies for the 
handling of patient health, such as patient care, community education, disease detection, public health 
monitoring, and referring patients [1]. 

At this time, health service industry has been progressing rapidly. However, the need for health 
services has not been fully met [2]. Adequacy of medical personnels and medical devices is a problem that 
requires serious handling. This problem causes the need to refer patients to better equipped health-care 
providers [3]. Referring patients from one health-care unit to other health-care units is called a referral 
system. A referral system is the organization of health services that regulates the delegation of duties and 
responsibilities of health services on both sides, vertical or horizontal. At this time, referral systems have 
made good progress, although they still face many problems. 

Increasing population increases the need for health services, such as the need for medical personnels 
and medical devices. At this time, most of the specialist practitioners practice in large hospitals that exist in 
big cities. This leads to other problems. The first problem occurs for patients who reside away from urban 
areas as they need more time and cost [4, 5]. The second problem is the difficulty to get a specialist according 
to the condition of the patient's illness [6, 7]. The third problem is that there are delays arising from sending 
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and queueing of patients at the general hospitals [8, 9]. The fourth problem is that, for hospitalized patients, 
patients must wait for up to one month or until rejection from the hospitals [3, 10, 11]. 

Besides, a doctor at the hospital often must perform re-examinations. This usually happens because 
the general practitioner at the primary health center often conducts referral without giving any initial 
examination, excludes the patient's medical record [12-14], and does not enclose the apparent reasons for 
referral [15]. These problems raise other problems, including: (1) the rejection by the hospital due to the 
incompatibility between the patient's illness with the physician's specialty, and (2) the patient treatment is 
delayed because the specialist has to perform re-examinations [7, 16]. 

In Indonesia, finding a referral hospital takes a long time: it takes more than 30 minutes to find the 
referral hospital. This is because doctors in primary health centers have to contact at least five hospitals to 
refer one patient [17]. Another problem that often occurs is the rejection of emergency patients by the 
hospital because of treatment room is not available [18], the referral hospital has no medical equipment, no 
specialist doctors, no referral letters from primary health centers [19], and patients cannot afford to pay the 
up-front payment [20]. Cipto Mangunkusumo Hospital (RSCM) receives more than 3,000 referral patients 
per day. Of these, only 30% of the patients go through the appropriate referral procedure [21]. The main 
cause of some of these problems is that referral systems in Indonesia have not functioned effectively [22, 23]. 

Some of the studies that have been done in the last few years provide solutions to overcome this 
problem of finding referral hospitals by developing an electronic referral system [7, 8, 16, 24-30]. The given 
solution is called e-referral system, which is an effort to improve the effectiveness of the referral process. 
However, most of the researches so far focused more only on referral communication, with the aim of 
facilitating the flow of patient referrals from one health-care unit to another. 

Researches conducted in [29-32] developed intelligent system models that take into account the 
appropriateness between patients’ health condition and the referral hospitals. Their researches resulted in 
systems that can be used to determine the referral hospitals by considering patients’ health condition. 
However, these researches have not taken into account patient's economic background. An excellent e- 
referral system should consider patients’ health condition and should also take into account patients’ 
economic background. By taking into account these two conditions, the problems that exist in current e- 
referral systems can be overcome. Our current research aims at developing a referral system model that takes 
into consideration patients’ health condition as well as economic background. The model is developed using 
Fuzzy Multiple-Criteria Decision Making (FMCDM) approach to find hospitals that fit the patients’ 
background. Furthermore, web service technology is used in the communication between primary health 
centers and referral hospitals. 


2. RELATED WORK 
2.1. Referral System Model 

The development of a referral system model that can determine referral hospitals based on patients’ 
condition is complicated [5, 26]. Determining the appropriate referral hospitals is influenced by many factors, 
such as patients’ geographical location, the criticality of the medical situation, transportation, distance, and 
physical condition of the patients [29], as well as socio-economic of the patients [33]. Research conducted 
by [29] developed a referral system that is able to determine the exact referral hospitals specifically for 
patients with heart disease. Parameters used related to the patients include age, sex, nationality, type of 
treatment, disease severity, and type of payment. Parameters related to the hospital include hospital type, bed 
capacity, regional status, JCAHO (accreditation) status, operating ability, AMI termination volume, and 
CABG capability. However, the developed system is limited to emergency heart disease, distance, and time. 
The developed system only used experts as a reference, so that at any time the system needs to be re-adjusted 
if there are changes. 

Several studies aimed at improving service quality in e-referral systems have been conducted 
in [28, 34, 35]. Their researches developed e-referral system models by replacing the use of telephone, fax, 
and reference request in hardcopy format into electronic one. The developed system model is already 
integrated into the Electronic Health Record (EHR) at several hospitals so that their units can easily access 
data. Unfortunately, their studies focused solely on switching hardcopy documents to electronics, hence, they 
have not been able to resolve current referral cases. 

Also, [36] and [4] added facilities to the e-referral system. That facility is the ability to determine 
whether a patient is referred or not and the facility for consultation between a general practitioner and a 
specialist in the hospital. The facility can prioritize referrals and delay referral times until referral hospitals 
are ready to receive referral patients. The study might provide a solution to the referred patient without initial 
screening, however, the study focused only on advising whether a patient is referred or not. Meanwhile, our 
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research adds a new facility to help general practitioner to choose the right hospitals based on patients’ 
background. 


2.2. Referral System Model Using Web Services 

Web service is a technology that can be used for communication between applications from 
different platforms so that applications can integrate each other. Web service technology has been widely 
used as a solution on e-business [37, 38]. One of the architectures for communication on web services is 
restful. Restful is an approach to web services that is easy to implement because it uses HTTP with rest 
principle. Restful is easy to implement, as simple as using HTML. Rest can also use XML or JSON to 
exchange data. Rest can differentiate messages because it analyzes the HTTP commands used during the 
request [17, 39, 40]. Besides, restful has main architectural aspects of addressability, statelessness, simple 
and similar interfaces, and multi-access [17, 39]. 

The research in [17] used web service technology to exchange information between primary health 
centers and hospitals. Web service technology is used to transmit medical record data to the hospitals and 
obtain information, such as availability of inpatient room, ICU room, specialist, and medical support 
equipment from the referral hospitals before sending a patient. 


2.3. Referral System Model Using FMCDM 

According to [41], Multiple-Criteria Decision Making (MCDM) is one way of making decisions 
based on analytical methods involving many criteria. A criterion is a value in the form of size, rule or 
standard used in the decision making. The problem in MCDM is the use of crisp sets because only small 
changes in a value lead to significant category differences. Therefore, MCDM is combined with fuzzy logic 
to solve the problem. Fuzzy MCDM can resolve the lack of precision in measuring the importance of criteria 
and alternative assessment of the evaluation criteria. FMCDM has been widely applied to address decision- 
making issues with multiple criteria and alternatives in a consistent, productive and systematic way [42-46]. 

Research that develops an intelligent system model that takes into account the appropriateness 
between the patients’ health condition and the referral hospitals is quite rare, one of which is performed 
in [30]. The study developed a model of a referral system called Multi-Agent Referral Decision Support 
(MARDS). The model is developed using MCDM to determine the most appropriate referral hospitals. Multi- 
Agent Provider System Selection technology is used to improve the quality of communication between 
general practitioners and hospitals. The fundamental difference of this research is the technology used for 
communication between primary health centers and hospitals. Also, the other difference is the criteria used. 
The current study adds insurance ownership criteria. Furthermore, this study also adds a facility to determine 
the type of treatment to patients. 


2.4. Parameters of E-Referral System 


The parameters used to determine referral hospitals in each country have different rules or policies. 
The differences can be seen in the previous studies in Table 1. 


Table 1. Parameters of E-Referral Systems in Previous Studies 








Study Country Disease(s) Parameters 
[29] USA Acute Myocardial Patient: age, gender, nation, type of treatment, disease 
Infarction severity, type of payment. 
(AMI). Hospitals: hospital type, bed capacity, regional status, JCAHO 


status (accreditation status), operating ability, AMI 
termination volume, CABG capability. 


[7] Canada All diseases. Symptoms of patients, specialists, time availability, location of 
specialists. 

[16] Canada All diseases. Gender, age, disability, general practitioner background, 
practice and community background. 

[6] Nigeria All diseases. Patient distance to hospital, patient's health condition, and 
hospital facilities. 

[27] Denmark All diseases. Age, gender, ethnicity, marital status, diagnosis and 


socioeconomic characteristics (such as income, education and 
employment status). 
[47] Not Cancer in the elderly. Patients: age, patient's willingness, patient residence, 
mentioned community, family, disease severity, clinical or biological 
signs. 
Specialist doctors: specialist type, specialty quality, CPG. 
[30] Ethiopia All diseases. Referral type, referral category, required services, hospital 
location, hospital level, service waiting time and particular 
referral type. 
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Table 1 show that seven studies discussing the search for hospitals or specialists for referrals in 
United States, Canada, Nigeria, Denmark, and Ethiopia. The diseases used for testing include acute 
myocardial infarction (AMD), cancers in the elderly, and all illnesses. Those seven studies were divided into 
three groups of parameters used, namely patients, doctors, and hospitals. Parameters related to patients 
include age, gender, type of treatment, severity of illness, type of payment, patient symptom, patient with 
special needs, community background, distance from patient to hospital, ethnicity, marital status, diagnosis, 
economic (education, income and employment status), comorbidity, patient's willingness, family willingness, 
clinical or biological signs, reference type, referral category, required services, and service waiting time. 
Parameters related to doctors include general practitioner's background, specialist type, and quality of 
specialization. Parameters related to hospitals include hospital type, bed capacity, regional status, hospital 
accreditation status, operational ability, AMI termination volume, CABG capability, and service waiting 
time. 

In this study, we use ten parameters: insurance receipt, hospital ownership, length of waiting room 
of ward, availability of medical support, availability of specialist doctors, poly handling time, waiting time 
for handling of accident and emergency department, ease of transportation, distance between patient’s 
residence and the hospital, and distance between the primary health center and the hospital. The parameters 
are used to determine referral hospitals that fit the patients’ condition. In addition, this study also uses 
parameters such as age, gender, blood pressure, weight, blood glucose, comorbidity, pulse rate, eye 
awareness level, motoric awareness level, oral awareness level, systolic blood pressure, persistent fever, 
sudden hearing loss or vision, sudden paralysis, active bleeding, evisceration or wound diseases, vital signs 
monitoring, intramuscular/intravenous antibiotics every eight hours, use of continuous or intermittent 
respirators at least every 8 hours. Those seventeen attributes are used to determine the type of medical 
treatment. This type of medical treatment is performed in a poly room, department of accident and 
emergency, ICU, inpatient wards, or outpatient. 


3. PROPOSED MODEL 

This research was conducted in Yogyakarta, one of the provinces of Indonesia. The facts were found 
as follows. The referral systems are mostly still done manually, i.e., searching for available and suitable 
hospitals is conducted by phone. 

The primary process in the current referral system consists of four sub-processes, namely: (1) Initial 
examination; conducted by medical personnels at a primary health center before referring a patient to a 
hospital. This preliminary examination is used to determine if the patient needs referring or not. (2) 
Determining the emergency level; which is the process to determine the severity of the patient's illness. The 
purpose of the process is to make it easier to determine the type of treatment provided at the referral hospital. 
Based in the type of treatment, the patient can be handled in poly, accident and emergency department, ICU, 
inpatient ward, or outpatient. (3) Searching of the hospital; which is conducted by medical personnels at the 
primary health center by phone. (4) Dispatching of a patient; if there is a hospital capable of handling, the 
medical personnels at primary health center will send the patient with referral documents. The referral 
documents can be a referral letter, lab results if any, and the result of the initial examination. The referral 
process from the primary health center to the hospital is depicted in Figure 1. 
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Figure 1. Referral process from primary health centers to hospitals 


Indonesian J Elec Eng & Comp Sci, Vol. 11, No. 2, August 2018 : 475 — 486 


Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 i) 479 


Based on Figure 1, we propose a new model for the e-referral system, which comprises five main 
parts: 

1. Interface; used by medical personnels at the primary health centers and the hospitals to interact with the 
referral system. The interface is developed using Java programming as Java supports multiplatform, is 
easy to develop, and dynamic. This part is illustrated in the Interface block in Figure 2. 

2. Probabilistic Neural Network (PNN); which is used to determine the emergency level of diseases. The 
classification method used in determining the emergency level of diseases is an artificial neural network. 
This part is illustrated in the PNN block in Figure 2. 

3. Web Service; which is used for searching referral hospitals. The searching process of the hospital is done 
by contacting many hospitals simultaneously. The service used is restful with the HTTP request method: 
Get, Post, Put, and Delete. JSON format is used in data transmission. In Figure 2, this part is illustrated 
by the Web Service block. 

4. Rules are used to determine the appropriate referral hospitals based on the patient's condition. The rules 
is generated from the currently applied rules. As an example, for the case of a diabetic patient, if the 
patient is a Jamkesda insurance holder and the type of care is through Poly and the patient requires 
inpatient wards, then the selected hospital is of type C or D with class III treatment room. In another case 
of a patient with diabetes, if the patient is PBJS class I insurance holder, the type of care is through 
accident and emergency department and inpatient wards, then the selected hospital can be of type C, D, 
B or A with class I, I, or I treatment room. 

5. FMCDM is used to provide a recommendation of referral hospitals to assist the medical personnel in the 
primary health center in the decision-making process. Fuzzy Multiple-Criteria Decision Making 
approach is used to provide the recommendation. The output is a list of hospitals that have been ranked 
based on the predetermined criteria. This part can be seen in the FMCDM block in Figure 2. 

Figure 2 shows that the proposed referral system model has these abilities: (1) contacting several 
hospitals and receiving information from several hospitals simultaneously; (2) determining the type of 
treatment before dispatching the patient; and (3) assisting medical personnel in the primary health center to 
determine the referral hospital sbased on the patient's background condition. The focus of this work is to 
develop a model for the hospital ranking process based on the patient's background, by creating a framework 
for selecting referral hospitals that match the patient's background using FMCDM. According to [48], 
FMCDM is performed using the following steps: problem representation, evaluation of fuzzy sets of 
alternatives, and determine the rank of all alternatives, which will be elaborated on in the following. 











ya Web Service 
\ (http request method: get, post, put, delete) ) 
(format data: JSON) 


Patient's data \ Ez 


: Hospital's data 








| p . | / G Rule Based i) 


= = 





Interface 


















List of referral 
hospitals 

















(A 
\ Criterion-3 } 




















\n 


| J : : = \Criterion-n) 





























Figure 2. The proposed model of the e-referral system 
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3.1 Problem Representation 

In this step, there are three primary tasks. The first task is to provide a list of alternative referral 
hospitals that is appropriate to the patient's background, based on specified criteria. If there are n alternatives 
(referral hospitals), then the set of alternatives can be denoted by A = {A; |i = 1, 2, 3, ..., n}. In this study, 
there are six alternatives, namely A = {RSU Permata Husada, RS Rajawali Citra, RSU Nur Hidayah, RSU 
PKU Muhammadyah Bantul, RSUD Panembahan Senopati, RSPAU Dr. Suhardi Harjolukito}. 

The second task is to determine the criteria used to rank the referral hospitals. If there are k criteria 
for determining the choice of some alternatives, then the set of the criteria can be denoted by C = {C; |i= 1, 
2, 3, ..., k}. In this study, there are eleven criteria, namely C = {acceptance of insurance, hospital ownership, 
duration of waiting time in inpatient ward, availability of medical support, availability of Specialist, duration 
of waiting time in Poly, duration of waiting time in accident and emergency department, ease of 
transportation, duration of waiting time in ICU, distance between the patient’s residence and the hospital, 
distance between the primary health center and the hospital}. And the next task is determining the framework 
model of the referral hospital selection that is appropriate to the patient's background. The framework model 
is depicted in Figure 3. 
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Figure 3. Decision-making framework of the e-referral’s hospital selection 


3.2 Evaluation of Fuzzy Sets 

At this stage, an evaluation of the weight of each criterion is performed. In this study, we use the set 
of the fuzzy triangle in determining the membership function of the weights of criteria and the degree of 
compatibility of each alternative. For example, the membership function of the weight of a criterion to 
determine distance is depicted in Figure 4. 

In determining the matching values, Equation (1), (2), and (3) are used: 
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(3) 


where x is the actual distance and u(x) is corresponding the membership function. 

Afterwards, the selection of the membership function for an alternative degree of compatibility is 
carried out. In this case, five linguistic variables are used, namely Very Less (VL), Less (L), Quite Good 
(QG), Good (G), and Very Good (VG). The weight of each variable is obtained from the fuzzy triangles, 
which can be observed in Table 2. 
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Table 2. The Linguistic Variable and Its 





Weight of Fuzzy Value 

Linguistic Weight of the fuzzy 
Far 5 

variable value 
VL (0.0, 0.0, 0.25) 
L (0.0, 0.25, 0.50) 
QG (0.25, 0.50, 0.75) 
G (0.50, 0.75, 1.0) 
VG (0.75, 1.0, 1.0) 





10 Distance (km) 


Figure 4. Membership function using fuzzy triangles to 


determine distance 


The linguistic variable is the used for the assessment of each criterion. After calculating the 
assessment of each criterion, the result as depicted in Table 3 is obtained. 


Table 3. The Linguistic Variable and the Assessment of Each Criterion 














Criteria Paneuisie Value Weight of the fuzzy value 
variable 

Cl Government (G) 1 d, 1, 1) 
Private (P) 0 (0, 0, 0) 

C2 Very Long (VL) (>16) (16, 19, 19) 
Long (L) (11 - 17 hours) (11, 14,17) 
Normal (N) (6 — 12 hours) (6, 9, 12) 
Fast (F) (1 —7 hours) (, 4, 7) 
Available(A) (0 — 2 hours) (0, 0, 2) 

C3 Long (L) (>25 minutes) ((25, 35, 35) 
Normal (N) (10 — 30 minutes) (10, 20, 30) 
Fast (F) (0 — 15 minutes) (0, 10, 15) 

C4 Long (L) (> 25 minutes) (25, 35, 35) 
Normal (N) (10 — 30 minutes) (10, 20, 30) 
Fast (F) (0 — 15 minutes) (0, 10, 15) 

C5 Very Long /Not Available(VL) (>16) (16, 19, 19) 
Long (L) (11 - 17 hours) (11, 14,17) 
Normal (N) (6 — 12 hours) (6, 9, 12) 
Fast (F) (1 —7 hours) (1, 4, 7) 
Available (A) (0 — 2 hours (0, 0, 2) 

C6 Not Available (NA) 0 (0, 0, 0) 
Available (A) 1 d, 1,1) 

C7 Not Available (NA) 0 (0, 0, 0) 
Available (A) 1 d, 1, 1) 

C8 Not Available (NA) 0 (0, 0, 0) 
Available (A) 1 d, 1, 1) 

C9 Very Difficult (NA) (Not Available /Others) (0, 0, 0) 
Difficult (D) (Public >= 2 times) (0.30, 0.30, 0.30) 
Easy (E) (Public < 2 times) (0.60, 0.60, 0.60) 
Very Easy (VE) (Private) d, 1, 1) 

C10 Far (F) (= 8 km) (8, 10, 10) 
Medium (M) (4 km — 10 km) (4, 7, 10) 
Near(N) (O km —- 6 km) (0, 4, 6) 

Cll Far (F) (= 8 km) (8, 10, 10) 
Medium (M) (4 km — 10 km) (4, 7, 10) 
Near(N) (0 km —- 6 km) (0, 4, 6) 
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As an example, a physician in a primary health center was requested to provide a weight for each 
criterion for the case of the referral patients handled at the ER and inpatient wards. Based on the linguistic 
variables in Table 2, the weighting result of each criterion for this particular physician is listed in Table 4. 


Table 4. The Weight of Every Criterion 





Criteria Rating of interest Weight of the fuzzy value 
Cl VL (0.0, 0.0, 0.50) 
C2 G (0.50, 0.75, 1.0) 
C3 VL (0.0, 0.0, 0.25) 
C4 G (0.50, 0.75, 1.0) 
CS VL (0.0, 0.0, 0.25) 
C6 VG (0.75, 1.0, 1.0) 
C7 VG (0.75, 1.0, 1.0) 
C8 VG (0.75, 1.0, 1.0) 
C9 VG (0.75, 1.0, 1.0) 
C10 L (0.0, 0.25, 0.50) 
Cll G (0.50, 0.75, 1.0) 





3.3. Ranking Alternatives 


The next process is to rate each alternative. Experiments are conducted using cases of patients with 
diabetes with the type of treatment through IGD and Inpatient Ward. After the input rating of each alternative 
matches each criterion, the next process is substituting the fuzzy number of each linguistic variable 


into (4), (5), and (6): 


i= (;) YG: aj), 
Qi = (;) Xe bi), 
Zi= (2) wee Ci), 


whose results are shown in Table 5. 


(4) 


(5) 


(6) 


Table 5. Assessment Indices for Every Alternative 


Match rating 





No. Alternative Cl @ © C4 CS C6 C7 C8 C9 C10 Cll Fuzzy match index 
1 Al P N L F VL A A A E N N (0.51; 2.00; 4.86) 
2 A2 P A L M VL A NA A_ VE N N (0.64; 2.00; 5.23) 
3 A3 P VL OL M VL A A A VE N N (1.37; 3.30; 6.77) 
4 A4 P N L F VL A A A E N N (0.69; 2.27; 5.36) 
5 AS G L L M VL A A A E N N (1.38; 3.30; 7.16) 
6 A6 G A L M VL T NA A E N N (1.41; 2.25; 5.80) 





After obtaining the fuzzy match index, the total integral is calculated by using (7): 


1%) = (3) (ac+b+ (1— aa). 


(7) 


The resulting total integral value is used to rank alternatives. The results of the total integral and the 


ranking calculations are shown in Table 6. 
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Table 6. The Total Integral Value of Each Alternative 


Total integral value 











No. Alternative a0 7 =0.5 EST 
1 AS 4.67 7.56 10.45 
2 A3 4.66 7.37 10.07 
3 A6 3.66 5.86 8.05 
4 A4 2.97 5.30 7.64 
a A2 2.64 4.93 7.23 
6 Al 2.51 4.69 6.86 





Based on Table 6, it can be concluded that alternative A5 is the alternative of the first order because 
it has the highest total integral value. Then, the next alternatives are A3, A6, A4, A2, and Al. 


4. RESULTS AND ANALYSIS 

To evaluate the proposed model, decisions generated by the model are compared to decisions taken 
by several experts. Experts in this case consist of medical personnels, such as doctors or midwives in primary 
health centers and hospitals. Likert Scale is used to assess the conformity of the model and the expert. 

For the evaluation, a questionnaire is designed with eleven questions with five possible answers, 
namely Very Appropriate (VA), Appropriate (A), Quite Appropriate (QA), Inappropriate (IP), and Very 
Inappropriate (VI). The score of each answer is set as follows: VI = 1, IP= 2, QA = 3, A=4, and VA =5. 
The questionnaire is distributed to ten experts at primary health centers and hospitals. Table 7 shows the 
experts’ answers to the questionnaire. Table 7 also indicates that the obtained conformity level (in 
percentage) is 81.45%, which is formed by the average of the total scores (in percentage) of each of the 
eleven questions in the questionnaire. 

The first question is about the facility to obtain the medical record data of the initial examination 
results. The purpose of the question is to know whether the data used is by the needs of the referral system. 
The results obtained are one expert answered IP, five experts responded A, and four experts answered VA. 
Thus, the conformity result of this facility is 84%. Hence, the facility can provide a solution for medical 
personnels of primary health centers to always perform initial examinations before sending patients to 
referral hospitals. 


Table 7. Questionnaire Result 








. Number of each response Total Percentage 
No. Question 
VI IP QA A VA _ score (%) 
1 Question | 0 1 0 5 4 42 84.00 
2 Question 2 0 0 0 6 4 44 88.00 
3 Question 3 0 0 0 4 6 46 92.00 
4 Question 4 0 0 1 7 2 41 82.00 
5 Question 5 0 0 1 7 2 41 82.00 
6 Question 6 0 0 2 57-3 41 82.00 
7 Question 7 0 0 1 8 1 40 80.00 
8 Question 8 0 1 0 9 0 38 76.00 
9 Question 9 0 1 0 8 1 39 78.00 
10 Question 10 0 1 0 8 1 39 78.00 
11 Question 11 1 1 0 6 2 37 74.00 
Average 81.45 





The purpose of the second question is to know whether sending preliminary examination data to 
several referral hospitals continuously can inform the patients’ health condition more quickly. The results 
obtained are six experts answered A, and four experts answered VA. The conformity result is 88%. The 
facility can solve the problem, so doctors at the hospitals do not have to conduct repeated examinations, and 
this can save time and cost. 

The purpose of the third question is to find out the time it takes to contact a referral hospital. The 
results obtained are four experts answered A, and six experts answered VA, with conformity result of 92%. 
It can be concluded that this facility can cope with contacting hospitals more quickly when compared to 
using the phone. 

The purpose of the fourth question is to find out whether the information provided by the referral 
hospital is appropriate to the needs of medical personnel at the primary health center. The results obtained are 
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one expert answered QA, seven experts answered A, and two experts answered VA. The percentage of 
conformity obtained is 82%. 

The fifth question is to know the criteria used to determine referral hospitals. The result obtained is 
that one expert responded QA, seven experts responded A, and two experts responded VA. The conformity 
result is 82%. 

The sixth question is to find out whether all the facilities in the developed model operate properly. 
The results obtained are two experts answered QA, five experts answered A, and three experts answered VA, 
so the conformity result is 82%. It can be concluded that the existing facilities already operate properly. 

The seventh up to the eleventh questions are to find out whether the resulting hospital 
recommendation conforms to the decision made by the experts. The five questions are for separate cases of 
patients. The first case is for patients with treatment through poly and outpatient. In this case, the conformity 
result produced by the developed model is 80%. The second case is for patients with treatment through poly 
and hospital wards. In this case, the conformity result produced by the developed model is 76%. The third 
case is for patients with treatment through emergency and emergency room. In this case, the conformity 
result produced by the developed model is 78%. The fourth case is for patients with treatment through 
accident and emergency department and inpatient wards. In this case, the conformity result produced by the 
developed model is 78%. The fifth case is for patients with treatment through accident and emergency 
department and ICU inpatients. In this case, the conformity result produced by the developed model is 74%. 
The results obtained is in the range of 74% to 76% because there is one expert who responded inappropriate 
and one expert answered very inappropriate. The result is low because the experts have concerns that the 
recommended hospital cannot be changed. It should not be a concern because the resulting output is a list of 
approved hospitals so that medical personnels can still select from the recommended list of referral hospitals. 

The ten experts who fill the questionnaire also provide a summary of the advantages of the 
developed system model as well as suggestions for improvement. The advantages of the developed model are 
that it is complete, according to current needs, and very helpful to solve existing problems. Suggestions from 
experts include the need for additional facilities to provide reasons for selecting the hospital, the need for 
support from the government, the need for confidentiality of patients’ identity, and the need for scheduling if 
there are outpatients. 


5. CONCLUSION 

The main capability of the developed model is to provide referral hospital recommendations based 
on the patients’ background. Recommendations are provided in the form of a list of referral hospitals based 
on FMCDM approach. Beside having this capability, the developed model is also equipped with facilities, 
such as modules to input preliminary examination data, to send preliminary examination data simultaneously 
to several hospitals, to request information from several hospitals simultaneously, and to determine the type 
of patients’ treatment when referred to hospitals. The types of treatment include treatment of outpatients 
through Poly, Inpatient Ward through Poly or Outpatient through IGD, Inpatient Ward through IGD, or ICU 
Inpatient through IGD. 

Based on the evaluation conducted, it can be concluded that the developed model conforms to the 
needs of e-referral. The conformity result of the developed model reaches 81.45% according to 10 medical 
personnels from primary health centers and hospitals. From the evaluation results, all the medics welcome 
positively the developed model for patients’ referral from the primary health centers to hospitals. Some of 
their suggestions are more technical, such as: lost connection, power failures, and the government’s 
involvement if it is to implement. 

The future work of this study includes methods to classify the severity of the patients’ disease. This 
classification aims to determine the type of treatment of patients when referred to the hospitals, so when the 
referral is conducted the type of treatment has been known. 
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